fix(lexer): stop dropping input after an unlexable byte - #3015

Summary

Lexer::lex silently drops all input from the first unlexable byte to the end of the string, so fix can delete part of the user's SQL and lint parses a truncated statement.

Root cause

In the lex loop, when lex_match stops at a byte that no matcher covers, the fallback is wired to the wrong string and always breaks:

let mut resort_res = self.last_resort_lexer.matches(str_buff);   // original buffer, not the remainder
if !resort_res.elements.is_empty() {
    break;                                                        // last-resort always matches -> always break
}
str_buff = resort_res.forward_string;                            // dead code
element_buffer.append(&mut resort_res.elements);

The last-resort matcher ([^\t\n.]*) can match the empty string, so resort_res.elements is never empty and the loop always breaks before reaching the "advance and append" lines. Whatever lex_match could not consume is discarded.

Concretely, select ф lexes to just select (the non-ASCII identifier is dropped). This does not panic, because the dropped bytes are always a clean suffix, but for a formatter it is silent data loss.

Fix

Run the last-resort matcher on the unlexed remainder res.forward_string, append its elements, advance str_buff, and break only when it makes no progress (guarding against an infinite loop). The unlexable run then becomes an Unlexable segment and lexing is lossless. Fully lexable input is unaffected, since the fallback only runs when lex_match hits a byte it cannot cover.

Testing

Added lex_preserves_unlexable_input, which lexes select ф through Lexer::lex and asserts the segments round-trip to the original text. It fails on main (left: "select ", right: "select ф") and passes with the fix. cargo test -p sqruff-lib-core is green (all tests) and cargo clippy -p sqruff-lib-core --tests is clean.

One honest note on my local run: the lib-dialects dialects fixture test reports a couple of expect test failed mismatches in my Windows checkout, involving \r\n in Snowflake udf_body fixtures. They are present on a clean main too (a CRLF/line-ending artifact of checking out on Windows) and this change does not add any; the Linux CI should be the real check.